Deploying deep neural networks (DNNs) on resource-constrained mobile devices presents significant challenges, particularly in achieving real-time performance while simultaneously coping with limited computational resources and battery life. While Mobile Edge Computing (MEC) offers collaborative inference with GPU servers as a promising solution, existing approaches primarily rely on layer-wise model partitioning and undergo significant transmission bottlenecks caused by the sequential execution of DNN operations. To address this challenge, we present Intra-DP, a high-performance collaborative inference system optimized for DNN inference on MEC. Intra DP employs a novel parallel computing technique based on local operators (i.e., operators whose minimum unit input is not the entire input tensor, such as the convolution kernel). By decomposing their computations (operations) into several independent sub-operations and overlapping the computation and transmission of different sub-operations through parallel execution, Intra-DP mitigates transmission bottlenecks in MEC, achieving fast and energy-efficient inference. The evaluation demonstrates that Intra-DP reduces per-inference latency by up to 50% and energy consumption by up to 75% compared to state-of-the-art baselines, without sacrificing accuracy.

The rapid advancements in machine learning techniques have led to significant achievements in various real-world robotic tasks. These tasks heavily rely on fast and energy-efficient inference of deep neural network (DNN) models when deployed on robots. To enhance inference performance, distributed inference has emerged as a promising approach, parallelizing inference across multiple powerful GPU devices in modern data centers using techniques such as data parallelism, tensor parallelism, and pipeline parallelism. However, when deployed on real-world robots, existing parallel methods fail to provide low inference latency and meet the energy requirements due to the limited bandwidth of robotic IoT. We present Hybrid-Parallel, a high-performance distributed inference system optimized for robotic IoT. Hybrid-Parallel employs a fine-grained approach to parallelize inference at the granularity of local operators within DNN layers (i.e., operators that can be computed independently with the partial input, such as the convolution kernel in the convolution layer). By doing so, Hybrid-Parallel enables different operators of different layers to be computed and transmitted concurrently, and overlap the computation and transmission phases within the same inference task. The evaluation demonstrate that Hybrid-Parallel reduces inference time by 14.9% ~41.1% and energy consumption per inference by up to 35.3% compared to the state-of-the-art baselines.

The use of artificial intelligence and machine learning is rapidly growing in a variety of industries, and with this growth comes the increased demand for high-performance computing resources. One option for powering these workloads is the use of GPU servers, which are specialized machines that are optimized for running complex computations quickly. However, the costs of using GPU servers can add up quickly, especially if you are running them for an extended period. Fortunately, there are ways to save money on GPU servers by taking advantage of the many options available on the cloud. In this post, we will explore some tips for getting started with GPU servers on a budget, and show you how to save big on cloud computing costs.

Deep learning has been widely deployed for online ads systems to predict Click-Through Rate (CTR). Machine learning researchers and practitioners frequently retrain CTR models to test their new extracted features. However, the CTR model training often relies on a large number of raw input data logs. Hence, the feature extraction can take a significant proportion of the training time for an industrial-level CTR model. In this paper, we propose FeatureBox, a novel end-to-end training framework that pipelines the feature extraction and the training on GPU servers to save the intermediate I/O of the feature extraction. We rewrite computation-intensive feature extraction operators as GPU operators and leave the memory-intensive operator on CPUs. We introduce a layer-wise operator scheduling algorithm to schedule these heterogeneous operators. We present a light-weight GPU memory management algorithm that supports dynamic GPU memory allocation with minimal overhead. We experimentally evaluate FeatureBox and compare it with the previous in-production feature extraction framework on two real-world ads applications. The results confirm the effectiveness of our proposed method.

Machine learning (ML) is an important part of modern data science applications. Data scientists today have to manage the end-to-end ML life cycle that includes both model training and model serving, the latter of which is essential, as it makes their works available to end-users. Systems for model serving require high performance, low cost, and ease of management. Cloud providers are already offering model serving options, including managed services and self-rented servers. Recently, serverless computing, whose advantages include high elasticity and fine-grained cost model, brings another possibility for model serving. In this paper, we study the viability of serverless as a mainstream model serving platform for data science applications. We conduct a comprehensive evaluation of the performance and cost of serverless against other model serving systems on two clouds: Amazon Web Service (AWS) and Google Cloud Platform (GCP). We find that serverless outperforms many cloud-based alternatives with respect to cost and performance. More interestingly, under some circumstances, it can even outperform GPU-based systems for both average latency and cost. These results are different from previous works' claim that serverless is not suitable for model serving, and are contrary to the conventional wisdom that GPU-based systems are better for ML workloads than CPU-based systems. Other findings include a large gap in cold start time between AWS and GCP serverless functions, and serverless' low sensitivity to changes in workloads or models. Our evaluation results indicate that serverless is a viable option for model serving. Finally, we present several practical recommendations for data scientists on how to use serverless for scalable and cost-effective model serving.

In early September 2020, NVIDIA debuted its second generation GeForce RTX 30 family of graphics cards, the Ampere RTX architecture. NVIDIA broke with tradition when its new generations of cards were sold more expensive than their predecessors, which means that the cost of training models has remained more or less the same. This time NVIDIA has set the price of new and more popular cards at the level of the previous generation of cards at the time of sale. For AI developers, this event is significant -- in fact, the RTX 30 cards open up access to performance comparable to the Titan RTX, but with a much lighter price tag. Data science developers now have the ability to train models faster without increasing costs.

Data centers that support AI and ML deployments rely on Graphics Processing Unit (GPU)-based servers to power their computationally intensive architectures. Across multiple industries, expansion in GPU use is behind the over 31 percent CAGR in GPU servers projected through 2024. That means more system architects will be tasked to assure top performance and cost-efficiency from GPU systems. Yet optimizing storage for these GPU-based AI/ML workloads is no small feat. GPU servers are highly efficient for the matrix multiplication and convolution required to train large AI/ML datasets.

According to a recent press release, "DeepBrain Chain is an Artificial Intelligence Computing Platform driven by blockchain. The DBC project is for global AI computing resource sharing and resource scheduling because many small businesses do not have the money to buy expensive GPU servers, but many companies have a large number of GPU servers which are idle. Scheduling global resources and increasing the utilization efficiency of resources are of positive significance regarding the AI business computing costs reduction. Its vision aims at providing a decentralized AI Computing platform, which is low cost, private, flexible, and safe. The DeepBrain Chain platform serves the interests of several parties. Firstly, the Miner's main income is rewarded with token from mining. Secondly, AI companies just pay small amounts to run. Thirdly, the Chain uses the smart contract to separate the data provider and data trainer physically. Thus, it protects the data of the provider."

SAN FRANCISCO--(BUSINESS WIRE)--DeepBrain Chain is an Artificial Intelligence Computing Platform driven by blockchain. The DBC project is for global AI computing resource sharing and resource scheduling because many small businesses do not have the money to buy expensive GPU servers, but many companies have a large number of GPU servers which are idle. Scheduling global resources and increasing the utilization efficiency of resources are of positive significance regarding the AI business computing costs reduction. Its vision aims at providing a decentralized AI Computing platform, which is low cost, private, flexible, and safe. The DeepBrain Chain platform serves the interests of several parties.

DBC is the first AI computing platform driven by blockchain. It is a new attempt between AI and Digital Currency. The company introduced its cloud platform in May 2017 and already created a working product with over 100 manufacturers using the platform including Microsoft, Samsung, Siemens, and Lenovo. DeepBrain Chain is an Artificial Intelligence Computing Platform driven by blockchain. The DBC project is for global AI computing resource sharing and resource scheduling because many small businesses do not have the money to buy expensive GPU servers, but many companies have a large number of GPU servers which are idle.